Search AI Products and News

In the field of deep learning, normalization layers are considered an indispensable component of modern neural networks. Recently, research led by Meta FAIR research scientist Zhuang Liu, titled "Transformer without Normalization Layers", has garnered significant attention. This research not only introduces a new technique called Dynamic Tanh (DyT), but also demonstrates the effectiveness of Transformer architectures even without traditional normalization layers.

Frustrated by the slow processing speed of large language models on long documents? Researchers from Tsinghua University have unveiled a groundbreaking technology – the APB parallel inference framework – that dramatically accelerates processing. Benchmark tests show this technology achieves a 10x speed improvement over Flash Attention when handling ultra-long texts. With the rise of models like ChatGPT, AI's ability to process vast amounts of text (hundreds of thousands of words) has increased significantly. However, this often comes at the cost of processing speed...

Against the backdrop of the global generative AI wave in 2022, YuanShi Intelligence (RWKV) completed tens of millions of RMB in angel round financing in December 2023, with investment from Tianji Capital. Following this financing, the company’s valuation doubled, and the funds will be used for team expansion, new architecture development, and product commercialization. The emergence of RWKV presents a strong challenge to the traditional Transformer architecture. With the advancement of large language models (LLMs), although the parameter scale of models is becoming increasingly large, issues like hallucination and accuracy persist.

Recently, Apple's research team and Meta AI researchers have jointly introduced a new technology called LazyLLM, which aims to enhance the efficiency of large language models (LLMs) in long-text reasoning. As everyone knows, current LLMs often face slow processing speeds when dealing with long prompts, especially during the pre-filling phase. This is primarily due to the modern transformer architecture's computational complexity, which grows quadratically with the number of tokens in the prompt

In the realm of artificial intelligence, the competition between open-source and closed-source models has never ceased. The recent release of Meta AI's Llama 3.1 model, however, seems to have drawn a watershed line in this contest. This is not merely the launch of a new model; it is a sign of the maturation of open-source AI, heralding the arrival of a new era.Llama 3.1, developed by Meta AI's team, is a new generation of large-scale language models. In over 150 benchmark tests, its 405B paramet

The Million Experts Mixture model proposed by Google DeepMind, a revolutionary study that has taken a significant step forward in the Transformer architecture. Imagine a model capable of sparse retrieval from a million mini-experts - doesn't that sound a bit like a science fiction novel plot? Yet, this is the latest research achievement from DeepMind. The core of this research is a highly parameter-efficient expert retrieval mechanism, which separates the computation cost from the parameter coun

Although the Transformer model is powerful, its efficiency issue during decoding has always been a headache. However, researchers from the Korea Advanced Institute of Science and Technology, LG, and DeepMind have given us a surprise this time—they have proposed a new Transformer architecture called Block Transformer, which directly boosts decoding speed by 10 to 20 times! But how did they achieve this? It turns out that they "chunked" the attention mechanism of Transformer. This completely over

In today's rapidly advancing field of artificial intelligence, a company named Etched is betting everything on an AI architecture known as Transformer. The company recently announced the launch of the world's first application-specific integrated circuit (ASIC) chip designed specifically for Transformers—Sohu—claiming that its performance far exceeds any GPU currently on the market and will bring revolutionary changes to the AI field.